Evaluating participating methods in image analysis challenges: Lessons from MoNuSAC 2020

Biomedical image analysis competitions often rank the participants based on a single metric that combines assessments of different aspects task at hand. While this is useful for declaring winner competition, it makes difficult to assess strengths and weaknesses participating algorithms. By involving multiple capabilities (detection, segmentation classification) releasing prediction masks provided by several teams, MoNuSAC 2020 challenge provides an interesting opportunity look what information may be lost using entangled metrics. We analyse results “Panoptic Quality” (PQ) used organizers, as well disentangled metrics detection, classification abilities algorithms separately. show PQ hides results, its sensitivity small changes in hard interpret these draw insights from them. Our also demonstrate necessity have access, much possible, raw predictions teams so can more easily analysed thus research community.